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(54) Method and apparatus for sensing object located within visual field of imaging device 



(57) An apparatus and a method is provided for 
sensing an object within an imaging field of an imaging 
device (121). The data of the reference background pic- 
ture taken within the imaging field is stored in a storage 
unit (124). The data of the picture taken by the imaging 
device (121) is read. The data of the reference back- 
ground picture is compared with the read picture data at 
each pixel so that the difference value between both of 
the data may be yielded. The difference value is com- 
pared with a predetermined threshold value at each 
pixel. Based on the compared result, binarization data 
about an image of the object is produced. Based on the 
binarization data, the object is sensed. Then, the data of 
the reference background picture stored in the storage 
unit (124) is properly updated with the new data. 
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(54) Method and apparatus for sensing object located within visual field of imaging device 



(57) An apparatus and a method is provided for 
sensing an object within an imaging field of an imaging 
device (121). The data of the reference background pic- 
ture taken within the imaging field is stored in a storage 
unit (124). The data of the picture taken by the imaging 
device (121) is read. The data of the reference back- 
ground picture is compared with the read picture data at 
each pixel so that the difference value between both of 



the data may be yielded. The difference value is com- 
pared with a predetermined threshold value at each 
pixel. Based on the compared result, binarization data 
about an image of the object is produced. Based on the. 
binarization data, the object is sensed. Then, the data of 
the reference background picture stored in the storage 
unit (124) is properly updated with the new data. 
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Description 

The oresent invention relates to a system for monitoring a picture which system uses an imaging device such as a 

the invention relates to a method and an apparatusfor sensing an object .n the sys- 
£S Soring a p^m whichmethod and apparatus are preferable in the case of needing so h,gh re.iab.Hty such 

ino SS^SSSSSZ requested a system for automatically sensing an object such as a person or a «m 
SEES!* a monitor based on a video signal independently of human eyes and issuing a predetern,ned repor «d 
aterm This type of automatic monitoring system is required to sense an object by processing a video s.gna. produced 



V A^me oTThe systems for sensing an object by processing such a video signal, there has beer iknowr .and w.defy 
used LTdiSe ence^ethod wherein the latest picture coming to the system in sequence ,s compared with , the one p» 
vSctur^ Ta difference of a luminance for each corresponding pixels is obtained, and an area w,th a large difference 

38 3 The tesicSow of the system for sensing a difference value will be described with reference to the f .owchart of Fig. 
8 As shown at a "Zo 81 a current picture is captured from a TV camera. At the next step 82, a d.fference of a lum,- 
na^ceTs S^Sl l pre-stored background picture and the captured picture. Theoat a step * b"""" 
prSess f is made for the difference value. At last, at a step 84. the digital value is compared w.th a predefined thresh- 

The otctuSThownT^eft hand of Fig. 9B is taken when one car 90 is coming into this crossing. The picture shown 

the background picture of Fig. 9A and the picture shown in the left hand of Fig. 9B. 

In thfs type of system, partial breaks or gaps 91 appear in an object to be sensed as >nd.cated in the picture shown 
in the right hand of Fig. 9B. These breaks or gaps may bring about an error in sensmg an object 

However this drawback may be overcome by applying a dilation and eros.on operation to the system. 

A Tarred example of a system for sensing a moving object by processing a difference for ^mple, a wrrtjng 
tWedls ^rSfrnTmS Handling with image Recognition Technique" written by Ueda. et. a.., has been pubhshed ,n 

"^Ty^^ 

. x- » ^kJL i n pin 1 1 at firc;t assume that the three continuous pictures 111, 112, 1 13 in time are 
ST !2^dS££I SEl wEXZZSi 1 2 and between the pictures 1 12 and 1 13 are calculated. 
tSKSim. values are made binarized and then are ^^^^^T^ 
as a result of which pictures 1 14 and 1 15 are obtained. Next, these binary images (pictures) 114 and 115 are ANDed 
for S common portions between these pictures, on which portions a picture 116 of the object 

In the aforementioned disclosed method, however, the following erroneous sens.ng may take place^That is. though 
an obleS i isroTSuaJ y a>ming into the view field, an object is erroneously sensed. For example, as shown m F.g^ 9C. 
H tht cSnge of fS2 or Sghting conditions leads to change of brightness of an overaH object to be imaged an 
d!Zl ^iroad) may be erroneously sensed though no object is coming into the view field. As shown ,n Rg 9D .f 
™ S«S ouTof the view field is made so luminous that a reflected ray 94 may take place in the area wrthin an 

LdnffiSdlht Slecti rayTs Z oneously sensed as the object 95. Further, as shown in Fig. 9E. if the shadow 96 
rSt of the field is falling on the area within the imaging field, the shadow may be erroneously 

^ BWhe wayliTaJerential calculus needs a reference background picture (for example, Fig. 9A). Hence, the refer- 
enc 3£££c picti re is created in advance and then pre-stored in a memory. To create the ref erence backg, ound 
Sre there has been known a system arranged to use a video signal (normally, a v.deo signal of one frame) taken 
En no ^s Sming into the view field as the reference background picture according to the operator's selection 
or I ^ systemSang^ toaverage a predetermined number of frames composing a picture at each pixel and to use the 

the M by chance, the noise is held in the background picture. It is thus difficult to obtam the exact background 
picturkFuS. from a view point of the noise circumstances in the transmission system, the m.ngl.ng of the no.se is 
rw cn rarA In the Dractical use, therefore, it is a great obstacle. 

In Z la£e sySem to obtain the exact background picture, it is necessary to make the number of frames compo - 
• rJofjnrP to be averaaed as numerous as 1 500 frames, for example. As a result, a large time gap (about 50 sec- 
ond ZlZ^ZZ ofsOH^Z^oe between a time when a picturefor creating the reference background 
Se is STanS time when a difference is processed for sensing the object. Hence, for example^ *™™*Z" 
is gradua ." made gloomier at dusk, for example, the time gap makes it impossible to create so exact a reference back- 
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ground picture as allowing the object to be sensed. 

It is an object of the present invention to provide a method and an apparatus for reliably and exactly sensing an 
object coming into a view field of an imaging device for a short time. 

It is a further object of the present invention to provide a method and an apparatus for sensing an object which 
method and apparatus bring about no erroneous sensing resulting from the adverse effect of disturbance factors such 
as change of a weather or lighting conditions, reflected light or shadow. 

It is a still further object of the present invention to provide a method and an apparatus for sensing an object which 
method and apparatus serve to positively distinguish any portion where a local temperature is changed within the view 
field of the imaging device from the other portions. 

It is another object of the present invention to provide a method and an apparatus for sensing an object with highly 
reliance by providing a method for automatically creating a suitable reference background picture to deriving a differ- 
ence value for a short time. y 

In the method and apparatus for sensing an object coming into the imaging field of an imaging device according to 
the present invention, the process is executed for storing in a storage unit data of the reference background picture 
located within the imaging field, reading the data of the picture taken by the imaging device, comparing the data of the 
reference background picture with the read picture data at each pixel, outputting a difference value of each pixel com- 
paring the difference value with a predetermined threshold value at each pixel, creating digital data of an object image 
based on the compared result, sensing if the object exists based on the binary data, and properly updating the data of 
the reference background picture stored in the storage unit. 

The other objects and advantages of the present invention will be apparent from the description that will follow with 
reference to the appended drawings. 
In the drawings 

Fig. 1 is a flowchart for describing an operation of an embodiment of the present invention; 
Fig. 2 is a flowchart for describing an operation of a further embodiment of the present invention; 
Fig. 3 is a flowchart for describing an operation of a still further embodiment of the present invention- 
Fig. 4 is a flowchart for describing an operation of another embodiment of the present invention; 
Fig. 5 is a flowchart for describing an operation of another embodiment of the present invention; 
Fig. 6 is a flowchart for describing an operation of another embodiment of the present invention- 
Fig. 7 is a flowchart for describing an operation of another embodiment of the present invention- 
Fig. 8 is a flowchart for describing an operation of sensing an object by using a differential calculus; 
Figs. 9A, 9B, 9C, 9D, and 9E are pictures for describing a problem caused in sensing an object by using the differ- 
ential calculus; 

Fig. 1 0 is a view for describing the effect of a similarity used in sensing an object according to the present invention- 
Fig. 1 1 is an explanatory view for sensing an object based on a difference in time according to the prior art- 
Fig. 1 2 is a block diagram showing a system for monitoring a picture to which an embodiment of the present inven- 
tion is applied; 

Fig. 1 3 is a flowchart showing an embodiment in which the present invention is applied to sensing an object comino 
into a railroad crossing; * 

Fig. 14 is a flowchart showing a process for creating a background picture according to an embodiment of the 
present invention; 

Fig. 15 is an explanatory view showing an example of an operation of sensing an object according to an embodi- 
ment of the present invention; 

Fig. 16 is an explanatory view showing an example of an operation of sensing an object according to an embodi- 
ment of the present invention; 

Fig. 1 7 is a flowchart showing an embodiment in which the present invention is applied to a system for monitorina 
a pyroclastic flow; y 
Fig. 18 is an explanatory view showing an example of a sensing operation executed by the embodiment in which 
the present invention is applied to a system for monitoring a pyroclastic flow; 

Fig. 19 is a flowchart for describing an operation of creating and updating a reference background picture according 
to a first embodiment of the present invention; 

Figs. 20A and 20B are diagrams for describing an effect caused by calculation of a median of pixel luminance val- 
ues; 

Fig. 21 is a flowchart for describing an operation of creating and updating a reference background picture according 
to a second embodiment of the present invention; 

Fig. 22 is a flowchart for describing an operation of a method for sensing an object according to an embodiment of 
the present invention; 

Fig. 23 is a flowchart showing an operation of an embodiment in which the present invention is applied to sensing 
an object coming into a railroad crossing; 
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Fias 24A 24B 24C and 24D are explanatory views showing an operation of sensing an object through the effect 
of the differential calculus according to an embodiment of the present .nventoon; 

Rg 25 fs a^chart for describing an operation of sensng an object according to another embod.men, of the 
^STSSmi for describing an operation of sensing an object according to yet another embodiment of the 
^!^££iry view showing an operation of sensing an object according to the embodiment of the 
present invention. 

■"ISSSiSK £ S«» in Fig* 1 » 7. M descry - be oriented ,o a synten, tor n»n»orina n 

of Figs. 1 to 7 is saved in the P^™™^ , *J inDUttinfl a picture a picture is read from a TV camera. Then, at a. 

value with a threshold value. corresponding variable threshold values. 

polarity of the difference to be positive. Or, if the change ^ y e be tive The variati0 n depends 

picture itself is made gloomier. th«s change causey *e p°lanty of the ^drffereroe rto oe neg ^ 

variation, so that the object may be exactly sensed. 

SS^ffiSSffi? SSSX S, . M * because the, correspond to „e stops 11 

* 1 H.^. 7 sSn,K;SSS* ^a,ono..K.^pic.une BW e,^enceb^ rou ndp 1 *n i ^^ 
Herein, tne similarity ^P re * e 1 that tne who | e i nput picture is made gloomier as compared to 
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20 



25 



object 106 hides the background. Hence, a low similarity takes place between the areas 104 and 106 

in the flow of Fig. 3, at the next step 35 to the binarization step 34, the similarity is calculated At a step 36 the cal- 
culated My ,s compared with the reference value. Depending on whether or not the similarly fs hfcher 'than ml 

s TZZT ae -TT as A t nd B havin9 ,he dmrent qua,ities from each other - that *• 

Th^ST? ""fT 18 Ch , an9ed ^ the ^ Where the 06,601 is located ' are ^ted fro-^ each other 

The calculation of the s.m.lanty at step 35 is executed only for the pixels at which the difference value exceeds the 

K^JSET" in the binarization pr " ess at the step 34 - This makes ■ possib!a to £ 

w Fia 2 X^oii^ 4 ;,! 0 ^" 8 '" 9 30 ° bieCt ' thS PrOCeSS ° f estimatin S the similarity is added to the process shown in 
10 Rg. 2. the polarity of the d.fference ,s positive, the binarization process is executed al a step 45 Then at a step 46 
the scanty of »• form of a local luminance distribution is calculated between the input picturTand the Telenet 
ground picture. At a step 47, the similarity is compared with the reference value. As shown in ^mSSSSSS 
an area 94 as be.ng brighter by light applied from a lamp, it is determined that the polarity of The direncVTs 
» E£ r' a ? '^ i9h - H6nCe ' 31 3 St6P ^ * iS n0t that any object SstsTn the ITSTSilSK 

15 sri h „srr the simiiarrty is not so hish 003 - R9 ' io) ' at a - 4 * * * d ™ 

That is. as indicated by the area 94 of Fig. 9D. in the area that is made brighter by light, though the luminance is 
Sr 9 ' PattemS ° P r ° ad ,eft " HenCe ' ^ the ,OCal areaS 104 and 105 ° f Fig. 10 the sSlSTm^e 

bright^™ 

Next, in the flow of Fig. 5, like Fig. 4, for sensing an object, the process of estimating the similarity is added to the 

K2L'2T n 9 ' 2 ° f ^ differenCe iS nS9atiVe ' 31 3 Step 55 ' the binarizatfor P roceireSc^!d 

At a step 56 the process .s executed to calculate a similarity of the pattern of a local luminance distribution between 
the input picture art the reference background picture. At a step 57. the similarity is compared with the reference va^e 

* !?« ' " 6XiStS an ^ 96 *" iS made dark bv the shad ™ * ^ Sect it s SSSSSSSt 
polarity of the drfference is negative and the similarity is high. Hence, at a step 58 it is not determined *mm»*!Z 
exists in the area 96 (shadow). Conversely, if the polarity of the difference is negative Z SS^T^JS 
at a step 59, rt .s determined that there exists an object in the area 96 similarity is not so high, 

30 thnul^hi 5 ', ar6a ^ dark by Xhe Shad ° W of the ob j ect itse,f - as indicat ed by the area 96 of Fig 9E 

s^ssns;.^ patterns on the road are ,eft - Like ,ocai areas 104 and 105 in R9 - 1 °> ^ * 

dar^^s^T^^' theref ° re ' " * distin ^ ish ^ *«• *• area 96 that is made 

In the flow of Fig. 6, there is introduced a process (step 67) of estimating a similarity of the pattern of a local lumi- 
nance distnbuton between the input picture and the reference background picture. This added pSSJJXZ- 
ble to sense an area where a temperature locally goes up. P 
At a step 61 an object is imaged by an infrared TV camera. The captured video signal is saved as the input picture 

oTth 9 e^Lt%^ m t' V^^ 3 ^ Pre - St0red in 1he mem ° ry for Sensina the diffe ™<* value between bl 

rtimSfh * Po'arrty. Then, at a step 63. the polarity data is stored in a memory. If. at a step 64, the polarityfe 
determined to be positive, at a step 65. the difference value calculated at the step 62 is compared wrth a pnSete mrimd 
reference value (threshold value) so that the binarization process may be executed. At a step 66 *e opeSion S 
cuted to ca culate the similarity between the reference background picture and the input picture. 5 at a step 6 T the sTm- 

up ,ow ' at a step 68, " is determined that there exists in the inpirt picture an a - To^Zl 

That is, a difference of a luminance distribution takes place between the area where the local temperature aoes ud 

z lz t er : rt ke r ? ble - The oi the form of the iuminance distribu « on b^SKSSTJS 

the reference background picture is made low. For example, assume that the method for sensing an object as shown^n 
F,g. 6 is applied to sensing a pyroclastic flow. If a lava dome is formed by the pyroclastic flow the temperature abmot Z 

EVZZ area of the lav l dome - Hence - the simi,arity is made ,OW ^ °" the oSSK^XfSS 

"e SfiXS' ^ Pyr0C,aStiC ,,OW ' tSmPera,Ure 9065 UP Uniform ' y ^ ^ Wh0 ' e CtoUdS 
In the flow of Fig. 7, there is introduced a process (step 77) of estimating a similarity of the form of a local luminance 
At a step 71 , an object is imaged by an infrared TV camera. The video signal captured by the TV camera is read 
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10 



15 



20 



25 



30 



35 



aa an ** p«» ft • — » « a aep 72. th, 

p,a-aj«l in a menxay is oomparao I «* « ^'^fTlJXS^ ■ ^orS «. a mantory. If M polarity la 
S, M pioturas and Ha polarity may b. an* v*e oaloulated at tha at* 72 „ oomparad 

temperature goes down in the input picture. hehween tne | OC al area whose temperature goes down 

That is, a difference of a luminance *M*on « ^^^^'^f ^ of a finance distribution 
and the local area whose temperature is inhabited to godown ^ The j^J* * « J ronsider ^ m e method 
is made lower between the input picture and ^^^^^£^V^ lava dome is fell down, the tem- 
tor sensing an object as shown in F,g. 7 is ^ h ^ 0 ^^^c^e ava do me is dropped. It means that 
perature of the lava dome is abruptly made because *e he pyroclastic flow, if the application 

nVe^Tscribed embodiments are arranged so that 

sys tem for sensing an obje* t such as a jjjjjar a -ft ^pr^uces a video signal repre- 

site where the camera can take a 'ookatthe whole ™° pressed for sensing an object. The description 

abr»r OT p ^ 

executed by the system shown in Fig. 12. according to an embodiment of the invention, 

of updating a background picture is added. oic ture a video signal picked by a camera 121 

»- 1«. » -ep.aa.nt, . J"*""^ £JJ££ ISSS^STtS ne* retetenoe bacKpround pieaa. .a am*. 

14 in parallel to the flow of Fig. 13. rurture are caotured from the TV camera at regular inter- 

Entering theflow of Fig^ 14, J^-^JJf^SS E5» mmes are saved (stored) in a memory. 

Va ™a™ 
for creating the background picture data r(x, y). 



40 



r{x,y) = medif U) (x,y)) (1) 

liis20 



45 



50 



55 



^Henoe.tbe releyenoe baoKpyoond p^ute ™^~^™%^gZL*. The. rame period 
,„ w. ambodtma* only 20 frames MM | pBW. « sam£d « ^^e*Lre on yrftioh the median oaloola- 

lt;Ko^ 

this matter later. if the creation of the background picture is terminated, at a step 143, the exist- 
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just created. Then, the operation goes to the next step 144. If the creation of the new reference background picture is 
not terminated, the operation goes to the step 144 of processing a difference. At a step 144, the input picture is divided 
into blocks each consisting of 2 x 2 pixels. The overall picture is divided into a block area f n (i, j) consisting of 160 x 120 
pixels. Then, an average luminance Af of each block area is derived, where n is a block number and i j represents a 
coordinate location of a pixel inside of the block 

Further, for the reference background picture data r(x, y). likewise, the overall picture is divided into a block area 
r n (i. j) consist.ng of 160 x 120 pixels. Then, an average luminance Ar of each block area is derived The average lumi- 
nances A, and A r of each block area are calculated by the following expression (2) 



is 



A r=jl[ Z f nV.J) (2) 



where B represents the subject block area. [B] represents a pixel number contained in the area B. (This embodiment 
20 uses a block consisting of 2 x 2 pixels. Hence, i, j<=B represents 0 § i s i and 0 § j s 1 and [B] is four ) 

Next, a difference value c n of an average luminance between the input picture and the reference background pic- 
ture is derived at each block by the expression (3). 



25 



c n = |Af-Ar| (3) 
(n-1,2,3...) 

At the succeeding step 145 of saving the polarity data, the polarity data S n of the difference is derived by the 
so expression (4) and then is saved in a memory. y 

s = ,1 (Af > Ar) 
n (Af * Ar) (4) 
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Next, at a determining step 146, the operation proceeds to the first or the second binarization step 147 or 152 
according to the polarity data S n . K 

These binarization processes are executed by comparing the difference value c n with a predetermined threshold 
value. The threshold value for the positive polarity S n , that is, the threshold value th+ at the step 147 is derived bv the 
expression (5). y 



th + = n + + K + 

where 



(5) 



2 ^ E (^-m + ) 2 



[« + ] 1 [R + ] 

where ^ represents a difference value between the input pixel and the background picture at the d-th block (see the 
55 expression (2)). 

H + is an average difference value of the blocks having positive difference values of the blocks predetermined to con- 
tain no object, this value representing an average level of noise contained in the picture, 

a + is a standard deviation of the difference values of the blocks having positive difference values of the blocks pre- 
determined to contain no object, this value representing how variable the level of the noise contained in the picture is 
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^ K + is a adjustab.e parameter and [FW] is a number of pixe.s contained in an area wherein the poiarity Sn is pos«ve 
^ SSTJSSU 152, likewise, the threshold vaiue ttr for the negative polarity Sn is derived by the following 
5 expression (6). 



, where 



10 



[R] 1 



75 



[A] 

where K' is an adjustable parameter and [R ] is a number of pixe.s contained in an area having a negative poiarity and 
"° °Atte steps 148 and 153 of calculating the similarity, for each block, the similarity Yn is ca.cu.ated by the following 
20 expression (7). 
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wher. D n represents an area near an n-th block and [ 1 -^^'-Z^ * ^Tare set as the three** val- 

SS pig ESSE. E-^JS^^SS^ ,„ ^— S- - — - - 

Uke«ise. at «re detenrtnint, a.ep .54. the ^'■'^S'JS. of me oSea iaaanaed 
156, it is determined that the object is sensed _ operates to turn on the alarm lamp 130 

the user that the object is sensed embodiment is capable of positively sensing only an object 

in Fig. 13 will be described wfth ref erenc^to Rgs ^ 1 \ neaf . raHroad crossing . The shadow 171B of 

In Fig. 15, on an input picture 171 at dusk, a car i * s determ ined by the conventional method, 
the car 171A extends to the inside of the 173A coming into the crossing, 

as indicated on a picture 173. rt .s ^^^^^if^^^^Z^^^P^ shown in Fig. 14 is executed 

mined. . *_ *v>« r** r 171 a hiriP<; a nortion of the background, the similarity is not 

in M - ™ ^TtSiSL S. .o Lually have an o*ot On 

££SS CSS F , g . 13. » is—d «« 1. ! "^j; « ~ ' S*£a „ d ^ ly dtettwWe d tan 



8 



EP 0 749 098 A2 

1 74B appears, it is determined that no object comes into the crossing. 

Next. Rg. 16 shows an input picture 181 where a ray of light 181 B applied by a head lamp of a motorcycle 181 A is 
intruded into the railroad crossing before the motorcycle enters into the crossing. In this case, if the input picture 181 is 
determined by the conventional method, as shown by the picture 183, it is erroneously determined that there exists an 

object 183A coming into the crossing. 

On the other hand, according to the embodiment shown in Fig. 13, at first, the process shown in Fig. 14 is executed 
to obtain a reference background picture 1 72 and update it to a new picture every about four seconds 

Next, since the polarity data S n is positive in the ray of light 1 81 B applied by the head lamp, the operation goes to 
the step 149 of determining the similarity. At this step 149, it is determined whether or not the similarity between the 
' latest reference background picture 1 72 and the input picture 1 81 is high. 

In the area where the motorcycle 181 A exists, since the motorcycle 181 A hides a portion of the background the 
similarity is not made so high. Hence, at the step 151 . it is determined that there exists an object. On the other hand in 
the area of the ray 181 B, since it does not hide any portion of the background, the similarity is made high As a result 
at the step 150, it is determined that it is not an object but a mere bright portion. 

According to this embodiment, therefore, as shown by the picture 184. the motorcycle 174A and the ray applied 
from rts head lamp are clearly distinguished from each other as the objects 151 and 150 shown in Fig 13 It means that 
only the object coming into the railroad crossing can be positively sensed. In this case, though the portion of the ray 
1 84B appears, it is determined that there exists no object coming into the railroad crossing 

Next, the description will be oriented to a method according to another embodiment of the present invention 
The arrangement of this embodiment is application of the present invention to monitoring a pyroclastic flow in an 
active volcano. In the apparatus for monitoring a picture as shown in Fig. 12, an infrared TV camera (infrared imaging 
device) is used for the camera 121 . The infrared TV camera yields the input picture on which a luminance is made larger 
in a warmer portion of an object. In the system shown in Fig. 12, the CPU 126 executes the process shown in Fig 17 
In Fig. 1 7. as depicted in brackets, the routines from steps 1 91 to 1 99 and from steps 201 to 203 are the same as 
those from the steps 141 to 149 and from the steps 152 to 154 shown in Fig. 13. Hence, these routines will be briefly 
described. The processing operation goes to any one of the step 199 of determining the similarity and the step 203 of 
the same function, depending on the luminance, that is. the polarity of the difference of the temperature between the 
input picture and the reference background picture. At the steps 199 and 203, the adjustable parameters T + and T are 
used as the threshold values for determining the similarity. If an area has a positive polarity and a similarity that is 
smaller than or equal to the threshold value, at the step 200, it is determined that the area (object) has a higher temper- 
ature. If the area has a negative polarity and a similarity that is smaller than or equal to the threshold value, at the step 
204, it is determined that the area (object) has a lower temperature. 

The operation of the method according to this embodiment will be described with reference to Fig 1 8 An input pic- 
ture 21 1 is a picture captured at an instance when a pyroclastic flow takes place. A numeral 212 denotes a reference 
background picture. 

Then, after the process of Fig. 1 7 is executed for the picture shown in Fig. 18, the resulting picture is made to be 
the picture 213. On this picture, a portion whose temperature is made higher by the pyroclastic flow is allowed to be 
positively distinguished from the portion whose temperature is made lower after the pyroclastic flow is flown out 

That is, since the difference of the luminance distribution pattern takes place between the area whose local tem- 
perature is higher and the other areas whose temperatures keep stable, the similarity of the form of the luminance dis- 
tribution is made lower. 

In the portion where a lava dome is formed by the pyroclastic flow, the local temperature abruptly goes up Hence 
he s.m. larity .s made lower. On the other hand, for example, in the clouds that are made warmer by the pyroclastic flow' 
the whole temperature uniformly goes up. Hence, the similarity is made higher. 

Hence, the addition of the step 199 of estimating the similarity makes it possible to clearly distinguish the area 
(pyroclastic flow) whose temperature goes up from the other area at the step 200 if there exist portions such as the 
heated clouds. 

Or. the other hand, in the area whose local temperature goes down, the difference of the luminance distribution pat- 
tern takes place between that area and the area whose temperature keeps stable. Hence, the similarity of the pattern 
of the luminance distribution is made lower. 

If the lava dome is fell down, it means that the heat source is dropped. The temperature of the relevant area is made 
abruptly lower. Hence, the area is made to have a lower similarity. On the other hand, in the area such as the clouds 
heated by the pyroclastic flow, if the heat applied by the heat source is reduced, the overall area is substantially uni- 
formly cooled down, that is, lowers its temperature. Hence, the similarity is made higher 

The addition of the step 203 of estimating the similarity, therefore, makes it possible to clearly distinguish the area 
whose temperature is made lower (pyroclastic flow is flown out) from the other area at the step 204 if there exist portions 
such as the clouds whose temperature is made lower. It means that the cooled area can be distinctly sensed 

In addition, it goes without saying that the present invention is not limited to the above-mentioned application but 
may be applicable to another system such as a system for monitoring an intruder to a prohibitive zone such as plant 
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ventiona. shortcoming of the erroneous sens.ng resultmg f rem ^S^of ihe environment such as 

illuminance. This thus makes it possible to lower ^^^^J^^^^ M the similarity of 
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Atlemards, if. at the step 303. the ? f * SHE Ton the picture signal to the stored 

a'KX'TMSSS S K--. as W e - each pixe, „ 
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value at the middle of the order. . . . „ M . _. h a - shown in Fia 20A the distribution as shown 

iS ' 3 ,rn?L a h n angeoHhe picture takes place over N frames, the luminance value of the same pixel of each frame is made 
San JL * In this case, the --^«^ L te ^^X- the stored N frames 
Then, assume that an .^S^JS!^^ "ramT'i pixels composing the object or the 
though the object or » * *S5SEn^ value from the pixels of the picture that does not 

pixels mingled wrth the « ■ «2SS. mSi^ue M Sefrom. By rearranging the luminance values of the pixels 

SStbothendsofthechara^ 

When the luminance values of the p.xels «^^*« ^ *?T ™ Slett hand (toward a value of 1) from the 

at least a half of a number N of stored frames. 



10 



75 



20 



30 



35 



45 



50 



55 



10 



EP 0 749 098 A2 

In principle, however, a moving object or noise is quickly moved or disappears. It means that the number of frames 
where the object or noise is caught is not so great. 

Hence, by calculating a median value, the influence of the moving object or noise may be restricted without having 
to read so many frames N. This calculation of the median value makes it possible to obtain a quite exact reference back- 
ground picture. In addition, the calculation of the median value just needs a half or less of the stored frames if an object 
15 Ca ^, 9 ^ ?u S ° me ° f fram6S - Hence ' by lowerin 9 a sampling rate, the frames to be stored may be reduced in number 
Next, the method according to another embodiment of the invention shown in Fig. 21 is arranged to add an object 
sensmg step 402 and a determining step 403 after the step 401 of receiving a picture. After it is determined whether or 
not an object ex.sts at the steps 402 and 403, the same process as described about the method of the embodiment 
' f„ hown ,n F ' 9 - 1 9 !? exeCLrted on, y for *«• P jcture th at ^ determined to have no object In addrtion, the description about 

™ t and 404 10 407 iS left 0Ut because those Presses are the same as those at the steps 

301 to 305. At the step 402. by using a differential calculus, it is determined whether or not an object comes into an 

!^! 9 M 9 l ,e ?' J • piCture is determined to ho* no object at the step 403 is stored in the memory at the step 
404. Next, if the number of stored frames reaches the predetermined number, the processes at the step 406 of calcu- 
■ labng a median value and the step 407 are executed to calculate a median value (step 406) and update a reference 
background picture (step 407). 

According to this embodiment, therefore, it is guaranteed at the step 403 that a new background picture used for 
updating the reference background picture at the step 407 does not have any object coming into the picture This thus 
makes it possible to greatly reduce the number N of frames required for calculating a median value to a small value such 
as N = 6 as compared to the value used in the embodiment shown in Fig. 1 9. 

Next, in the method according to another embodiment shown in Fig. 22, the processes done by the embodiments 
shown in Figs. 19 and 21 are allowed to be executed in parallel. At the steps 305 and 407, that is, two-system processes 
of updating a reference background picture, the method of this embodiment is capable of constantly yielding the latest 
reference background picture. 

♦h l ? h th ^ r r b ° dim ! n l Sh ° Wn in RQ - 1 9 ' the number * frames t0 be stored is about 300 - wh 'cn is far more numerous 
than that of the embodiment shown in Fig. 21. Hence, the interval for updating the reference background picture is 
made considerably wide. It means that the freshness of the reference background picture is not necessarily satisfactory. 

On the other hand, in the embodiment shown in Fig. 21 , if yes at the step 403 of determining whether or not an 
object exists, the process at the step 407 of updating a reference background picture is not executed. In this case it is 
possible to make no use of such a special advantage as reducing the necessary number of frames 

The method according to the embodiment shown in Fig. 22, however, can make use of both advantages of the 
embodiments shown in Figs. 1 9 and 21 and allow the processes for these embodiments to use the corresponding latest 
reference background picture. Hence, the reference background picture can be made more exact 

Fig. 23 shows a series of processes from a process of creating a reference background picture to a process of 
sensing an object which processes are executed by the CPU 126 in the system shown in Fig. 12. This embodiment is 
based on the method shown in Fig. 19. 

At a step 501 of reading a picture, a video signal is read as picture data consisting of 320 x 240 pixels from a TV 
camera 121. Then, at a step 502 of storing a picture, the picture data is stored in a picture memory 124 

At a determining step 503, if the number of the frames being stored reaches a predetermined value N, (for exam- 
ple, Ni = 300 = N). the operation is branched into a step 504 of calculating a median. If it does not reach the value N1 
the operation is branched into a step 506 of sensing an object. 

At the step 504 of calculating a median, the process is executed to rearrange the stored N frames as shown in Fig 
20 for each pixel and calculate a median value M by the following expression (8). 



x ix.y) = med [f {i) (x,y)) (8) 



where f«(x, y) represents the picture data composed of 300 frames and i takes a value of i = 1 2 300 In addition 
the expression (8) is the same as the case that the number N of the frames in the expression (1) is 300 

At the step 505, the median value derived at the step 504 is stored in a storage area for the reference background 
picture allocated to the picture memory 1 24. 

Next, at the step 506, the process is executed to derive a difference between the picture data read at the step 501 
and the reference background picture stored in the storage area of the picture memory 124. If the picture data has a 
portion with a larger difference than a predetermined threshold value, the portion is sensed as an object 

At first, the reference background picture as shown in Fig. 24A is compared with the picture containing an object to 
be sensed as shown in Fig. 24B. At each pixel located at the same coordinates of both of the pictures, the difference of 
the luminance is derived. The picture composed on the difference value is made to be as shown in Fig 24C Then the 
difference value is given a predetermined threshold value and then is made binarized. As a result, the picture shown in 
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Fi9 SSS* . - -*» — ■ »«- * «»" - me — " Hence - onl¥ * e * a ' ,0,,ed 

to be sensed. . fe b ^ on tne resu |t gi ven at the step 506. If any object 

,he TV samara 121 to*. picMe background pMuto Is srsstsd .t a step 612. *s oow one is 

S^l iJ. N, *s opsrs.,00 * branched to . ^SS^SSSK^-W*. P*-— ouoiber N e 
|8> ' Toooisdiaov^osovsoatioestepelt issfcrsd io «.*. ,*rencs badrgro.id pttureallocoted 

'SSfeMSK- , 2, Hs^. *a*^ 
about these processes is left off the specif .cation. _ executed This is the same as the process of the 

in succession, at a step 706. the process of "^J^^liS ™ * is the same as the process of the step 
step 501 . And. at a step 707, the process of sensing an ob ^ jrst system of process is the same 
506 Then, a first and a second systems of Smets *e Shod shown in Fig. 21 . 

as the method shown in Fig. 19. The second are the same as the processes of 

The processes of steps 708 to 71 0 corresponding to ^ef ^syst^ ot proces^ di to the second 

the steps P 502 to 504 shown ^J 8 ^^ * Sence 'he description 

35 system of process are the same as the < ™ 7 ™! p Dicture obta ined by the first system of the process .s 

rrsTcfsr^^^ 

^ process is taKen as a candidate B of the referee I — Jg^SSTlrt. one of a reference background picture 
At steps 718. 719 and 720. the processes are executed to ^^Swound picture B obtained at the ste P 716 
A obtained at the step 71 1 for the first '^^^^^SS^ stored in the memory with the 
for the second system of process and replace the previous r ^^^ fram ^ read attne ste p 709. and a number 
selected one. A number N 0 of frames read at the step 703 a <* re)atjon * Nq > N , > m 2 is set. a 

a recommended value of N 0 is N 0 _ 500. embodiments with reference to Fig. 27. 

ground picture are reduced in number, the exact reterence v . w j,, unlinanC e. the latest refer- 
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In addition to the above-mentioned applications, the present embodiment may be applicable to a system for moni- 
toring an intruder to plant facilities or prohibitive zones. 

According to the present embodiment, the calculation of a median is used for creating the reference background 
picture. Hence, the method of this embodiment can overcome the drawback the conventional technique of creating the 
reference background picture has involved and is endurable in light of noise. Further, the method of this embodiment 
needs a smaller number of frames for creating the reference background picture as compared with the method 
arranged to take an average of each pixel therefor. Hence, the method of this embodiment offers a highly reliable mon- 
itoring system with the latest reference background picture. 

In addition, an object sensing method according to the present invention may use luminance data as well as color 
data of the video signal. 

Claims 

1 . An apparatus for sensing an object within an imaging field of an imaging device, comprising: 

storage means (124) for storing data of a reference background picture fitted in said imaging field; 
means (1 23) for reading data of a picture taken by said imaging device; 

means (125, 126) for comparing the data of said reference background picture data with said read picture data 
at each pixel and outputting a difference value of each pixel; 

means (126) for updating the data of said reference background picture saved in said storage means; 
means (126) for comparing said difference value with a predetermined threshold value at each pixel and pro- 
ducing binarization data about said object based on the compared result; and 
means (126) for sensing whether or not said object exists based on said binarization data. 

2. An apparatus as claimed in claim 1, wherein said means for outputting said difference value contains means, for 
sensing a polarity of said difference value and said means for producing digital data contains first binarization 
means having a threshold value for a positive polarity of said difference value and second binarization means hav- 
ing a threshold value for a negative polarity of said difference value. 

3. An apparatus as claimed in claim 2, further including means for comparing the data of said reference background 
picture with said taken picture data at each block consisting of a predetermined number of pixels and calculating a 
similarity of said data between the blocks of said both of data, and said means for sensing whether or not said 
object exists contains means for sensing intrusion of said object based on said similarity and said binarization data. 

4. An apparatus as claimed in claim 3, wherein the data of said reference background picture and said current taken 
picture data contain data about luminance of plural pixels composing said picture. 

5. An apparatus as claimed in claim 4, wherein said imaging device includes a television camera (121). 

6. An apparatus as claimed in claim 4, wherein said imaging device includes an infrared imaging device. 

7. An apparatus as claimed in claim 1, wherein said means for reading said data operates to read said picture data 
having a predetermined number of frames from said imaging device at a predetermined period, said means for 
updating the data of said reference background picture contains means for calculating a median of said read pic- 
ture data composed of the predetermined number of frames at each pixel and operates to replace the data of said 
reference background picture saved in said storage means with said median value of each pixel as the new data of 
said reference background picture. 

8. An apparatus as claimed in claim 7, wherein said predetermined number of frames read from said imaging device 
by said means for reading the data is smaller than the number of frames output by said imaging device at said pre- 
determined period. 

9. An apparatus as claimed in claim 7, wherein the period at which said data reading means reads frame data from 
said imaging device is longer than the frame period of output of said imaging device. 

1 0. An apparatus as claimed in claim 7, wherein the data of said reference background picture and the data of said 
taken picture contain data about luminance of plural pixels composing the picture. 
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An apparatus as claimed in claim 7, wherein said means for sensing whether or not said object exists contains 
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means (1 30) for issuing an alarm if said object is sensed and said updating means operates to update the data of 
said reference background picture if said sensing means does not sense said object. 

„». ^ « rbimpH in claim 1 1 wherein said means for reading the data contains first data reading means 
£ and second data reading means for reading picture data com- 

of saidfead ^ daS composed of the N1 frames at each pixel and second median ca.culat.ng means for ca«- 
cuSino a median value of said read picture data composed of the N2 frames at each pixel and 

saTSSng means operates to replace the data of said reference background picture saved in sad stor- 
age mine wfh themedian value of any one of said first and second median emulating means as the new data of 
said reference background picture. 
13. A method for sensing an object within an imaging field of an imaging device, comprising the steps of ; 

updating (143) data of a reference background picture within said imaging field, said data stored in storage 
means; 

rpadina(141 1 42) data of a picture taken by said imaging device; 

comparing , J 2) the data of said reference background picture stored in said storage means with sad read pic- 
ture data at each pixel for calculating a difference value (cn) of each pixel; 

Si^flT 1C> said difference value with a predetermined threshold value at each pixel and producing 
binarization data about the picture of said object based on the compared result: and 
sensing (17) whether or not said object exists based on said binarization data. 

« id A method as claimed in claim 13. wherein when calculating said difference value, a process is executed to sense 
aTotentt oTsad oSrence value and at the step of producing said digital data, a process ,s executed to use a , * st 
(thT«*e pdartty is positive (15) or a second threshold value (th") if the polanty ,s negative (16). 

A method as claimed in claim 1 4. further including the step of comparing the data of said reference background pic- 
ture wTtS aata oisaid taken picture at each block consisting of the predetermined number of pixelsand cateu- 
SgTslmilaniy fan) of the data va.ue between both of said data blocks based on the compared result, and ,n 
which said object is sensed on said similarity and said digital data. 

A method as claimed in claim 15. wherein said step of calculating the similarity is located after said step ol f a^bng 
t££Z date Id the similarity is calculated only for the pixels whose difference values depart from said threshold 

value. 

-.7 A method as claimed in claim 16. further including a step of comparing said calculated similarity with a predeter- 
min« .Tete^e^ue and at said step of sensing the object, it is determined that said object ex,sts rf said simHanty 
40 is higher than said reference value. 

a method as claimed in claim 1 7, wherein the data of said reference background picture and the data of said taken 
oicTre ctntain ^ dSr a ^uminance of plural pixels composing the picture, said object similarly is a value for rep- 
r S Tmila^SSrLtion pattern of local luminance values of said picture, and at said step of sensing 
Sd o^ect thTproceS fs executed to determine as existence of no object an area having p*e.s whose difference 
values have a positive polarity and whose similarities are higher than said reference value (48). 

a method as claimed in claim 1 7 wherein the data of said reference background picture and the data of said taken 
iTureSnSun M taTZitano. of plural pixels composing the picture, the similarity of said object is a value 
fofreprSeSng ^S^fmii^7of a distribution pattern of local luminance values of said picture, and at : «d*p > of 
Sect a process is executed to determine as existence of no object an area having Panose drf- 
ference values nave a negative polarity and whose similarities are higher than said reference value (58). 

A method as claimed in claim 16. further including a step (36) of comparing said calculated similarity with a prede- 
ter^SS Preference value and steps (37. 38) of distinguishing an area having pixels whose similarities are higher 
thLnsafd reference va iue from anarea having pixels whose similarities are lower than said reference value. 

A method as claimed as claim 20, wherein said imaging device is an infrared imaging device, the date of saidref- 
erenc! ^Sckg^nd picture and the data of said taken picture contain data about luminance of plural pixels com- 
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posing the picture, the similarity of said object is a value for representing the similarity of a distribution pattern of 
local luminance values of the picture, and at said step of distinguishing said areas from each other, a process is 
executed to determine as an area having a higher temperature than another area an area having pixels whose dif- 
ference values have a positive polarity and whose similarities are smaller than or equal to said reference value (68). 

22. A method as claimed in claim 20, wherein said imaging device is an infrared one, the data of said reference back- 
ground picture and the data of said taken picture contain data about luminance of plural pixels composing the pic- 
ture, the similarity of said object is a value for representing the similarity of a distribution pattern of local pixel values 
of said picture, and at said step for distinguishing said areas from each other, a process is executed to determine 
as an area whose temperature is made lower than the other areas an area having pixels whose difference values 
have a negative polarity and whose similarities are smaller than or equal to said reference value (78). 

23. A method as claimed in claim 13, wherein at said step of reading the data, said picture data composed of the pre- 
determined number of frames is read from said imaging device at a predetermined period, and said step of updat- 
ing the data of said reference background picture contains a step (304) of calculating a median (M) of said read 
picture data composed of the predetermined number of frames at each pixel and a step (305) of replacing the data 
of said reference background picture stored in said storage means with said median value of each pixel. 

24. A method as claimed in claim 23, wherein at said step of reading the data, said predetermined number of frames 
read from said imaging device is smaller than the number of frames output by said imaging device at said prede- 
termined period. 

25. A method as claimed in claim 23, wherein at said step of reading the data, the period at which the frame data is 
read from said imaging device is longer than the period of the frame output by said imaging device. 

26. A method as claimed in claim 23, wherein said step of sensing the object contains a step of issuing an alarm if said 
object is sensed and at said updating step, a process is executed to update the data of said reference picture if said 
object is not sensed. 

27. A method as claimed in claim 23, wherein said step of reading the data contains a first data reading step (603) of 
reading picture data composed of N1 frames and a second data reading step (610) of reading picture data com- 
posed of N2 frames that is less than said N1 if said object is not sensed (N1 and N2 are both a positive integer of 
2 or more), 

said step of calculating the median contains a first median calculating step (604) of calculating a median of 
said read picture data composed of the N1 frames at each pixel and a second median calculating step (61 1 ) of cal- 
culating a median of said read picture data composed of the N2 frames at each pixel, and 

at said step of updating the data, a process is executed to replace the data of said reference background 
picture stored in said storage means with a median value derived by any one of said first and second median cal- 
culating steps as the data of said reference background picture. 
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